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ABSTRACT 

The efforts of the Austin (Texas) Public Sci.'^ol 
District to improve the readability and usefulness of technical 
reports are reviewed. Coincident with, or consequent to, a 1977 
letter by W, E. Jones to the editor of an Austin paper (the "Austin 
American Statesman") complaining about excessively lengthy (500-page) 
evaluation reports, the district began to separate technical reports 
and summary documents. Technical reports becaune in-house documents 
and final reports tooJc on a more newsy reporting style, with more and 
simpler graphics, simpler language, suggestions of actions rather 
than direct recommendations, standardized report features, an 
executive summary, and design features to guide the reader to 
important elements. Positive features of an evaluation report should 
be; (1) succinct description of the study; (2) succinct description 
of the analysis, in an attachment rather than in the text; (3) 
succinct description of the results; (4) conversational language; (5) 
simple presentation of numbers; (6) single idea tables; (7) placing 
complex tables in attachments; (8) simple, single-idea graphics; (9) 
popular format; (10) attractive report covers; and (11) a descriptive 
or inspiring title. Attachments Include four pages of before and 
after reports reflecting the new policies. (SLD) 
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The letter to the editor: W« E Jones is a physician in Austin, 
with an office on one of the major streets we travel. Each tinie 
weseehissignalcmgthecurbitisareniinderofhis 1977 
lettw.\ 

To the credit of the Office of Research and Evaluation staff, 
our reaction to his lambasting was positive. We made an office 
picture of us all standing in front of the newspaper clipping, 
pointing our fingers at each other ^ and even commissioned 
cartoons about The Wonderful Report" The positive reaction 
was not that we were amused rather than irritated, but that we 
actually took Dr. Jones' message to heart Now, do not get the 
impression that we wholeheartedly agreed with his conunents. 
We disagreed mightily witii his sentiment that the newspaper 
article had adequately reported the achievement test results in 
just a few paragraphs. 

The message we received and follow today is that our public 
(including school board members) do not have die time, inter- 
est, or inclination tc read nxne than a few paragraphs— much 
less a full-length, carefully crafted, research-style report The 
major flaw in the letter's logic, tiiat it is possible tt) oMidense 
the salient points into a few paragraphs, is evident jo anyone 
who has attenq)ted tc write a report about student achievement 
test results. Each answer given raises several mare questions 
tnat require more detail. The bottom line is that evaluators 
must answer as many questions as possible in the report, but 
provide additional data for other persons to study further. 

Our ongouig battle is with headlme writers. The local edu- 
cation reporters consistendy claim that someone else writes the 
headlines. Those headline writers epitomize the negative 
nature of news reportin^-nnamely, bad news sells papers. A 
nK)re insidious problem with the news reporters is the pressure 
they represent to reduce all issues U) a single number that can 
be higMighted, compared, and used as the bottom line in 
judging anything. In a larger sense, our elected officials and 
even our top administrators want to reduce all issues to a single 
statistic, a simple measure of a tremendously complex issue. 
This usually cannot be done, but often is attempted by selecting 
one from a number of key statistics. 

This paper is intended to advocato relegating indulgence in 
description and analysis to technical reports, and avoiding 
oversimplification of evaluation summary reports. In so doing, 
we will present options for reporting at different levels to meet 
the needs of a range of audiences. 
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After the pubUcadon of Dr. Jones' letter, but not enttiely as 
a result of it, the following key trends began to be evidenced in 
our reporting. 

L Technical reports were completely separated from the 
shorter summary documents. 

2. Technical reports became in-house documentation of 
data collecaon, storage, and analysis procedures. 

3. Fmal reports, as the shorter reports are now called, took 
on a nxm newsy reporting style. Charactmstics of this 
style are now evident 

• More graphics 

• Simpler graphics 

• Huge tables with hundreds of numbers nK>ved to the 
back of reports as attach nsents 

• Technical terms avoided or defined 

• Laborious conversicm of numbos fiom a table or 
graph into text replaced by statements of trends or 
key numbers 

• Key findings listed at the beginning of sections of 
the reports 

4« Findings were written tc suggest action that needed to 
be considered — stopping short of making direct recom- 
mendations. (More cm this issue later.) 

5. Report features such as a table of contents, executive 
summary, and definitions wr^e standardized, so a 
reader coukl begin to look for them in the same pkce in 
all reports* 

6. An executive sunmary was included. 

7. Headings, boxes, bullets, and other organizational 
features began to guide the reader to important ele- 
ments« 

Overall, either coineidentally or consequently, the letter to 
the editor ushered in a time of tremendous restructuring of our 
evaluaticMi reports; a trend that reflected, or even impacted, tt> 
some degree, how evaluation reporting has evolved across the 
country. 
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Recommendations 



Over the years, there has been a fascinating debate about 
the appropriateness of recommendations in an evaluation 
report. Locally, we began in 1973 avoiding any hint of a rec- 
ommendation, believing strongly that the decision makers and 
the program stafif have to decide on options and select from 
among them. During this time, there were other active offices 
of research and evaluation that were maldng clear and direct 
rccommicndations for action in their reports. Our history of 
recOi^imendations moves from ''none offered" in the 1970s to 
an attempt to wri»«; key findings that require action in words 
that call for action, and even make diat action clear. After an 
internal debate, the staff decided in the late 70s to divide major 
fmdings into two categories: 

1. Major Positive Findings and 

2. Major Findings Requiring Action. 

In about 1986, this dichotomy was dropped in favor of a single 
heading: 

Major Findings. 

This was partly a result of discovering that some programs had 
no major positive outcomes, and others had jutco^ncs that were 
neutral. 

From observations of the winning entries in AERA Divir>ion 
H's outstanding report competitions across the years, there is a 
definite tendency for the judges to select reports with strong 
recommendations — or at least direct statements about changes 
that are needed. The styles and range of recommendations that 
are evident in the reports entered in the AERA Division H out- 
standing publications competition across the years are summa- 
rized at the ri^Uit 

The major caution that must be heeded whenever an evaluator 
makes a recommendation is whether or not the evaluator 
creates a situation in which his or her ownership in ±at recom- 
mendation biases future objectivity in evaluating the impact of 
any changes made— or even objectivity toward a recommended 
change that was not made. We still tend to shy away from rec- 
onmiendations that might create such ownership. Our prefer- 
ence remains to lay out options that are available for selection 
or iO merely point out a situation that calls for change of some 
kind. The easy way out? Possibly, but often it is more difficult 
to describe several options than to grasp an obvious one that 
has been incc'mpletely considered. 



LEVELS OF RECOMMENDATIONS 

1. No recommendations; r lajor findings stated 
matter-of-factly as descriptions: 

The achieyement gains of students attending summer 
school were equal to, but no greater than those of 
similar students who did not attend summer school. 

2. Categorizing flndings to highlight those that 
require action: 

MAJOR FINDINGS REQUIRING A^ION : 
The achievement gains of students attending sur/uner 
school were equal to, but no greater than those of 
similar students who did not attend suncner school. 

3. Stating findings that require action in terms 
that iiidicate the action: 

Without revision, the summer school curriculum nnd 
schedule are not effective in producing achieve' Aent 
gains for summer school attenders greater than the 
gains made by nonatterders. 

4. Stating options that should be considered: 

The achievement Qoins of students attending summer 
school were equal to, bu: no greater than those of 
similar students who did not attend summer school. 
Options that should be considered include lengthening 
the summer school term; making structural changes 
such as matching the curriculum more closely to the 
regular year curriculum, eliminating field trips and 
other activities that are not direct teaching, and 
matching summer teachers with their regular students; 
or eliminating the summer sessions and redirecting 
the funds to regular year aclivities that have proven to 
be effective, 

5. Recommending a specific action to be taken: 

The achievement gains of students attending summer 
school were equal to, but no greater than those cf 
similar students who did not attend summer school 
Because summer school has consistently proven to be- 
ineffective, summer school should be ended, mi funds 
should be redirected to regular activities that have 
proven to be effective. 



Readers wfll vary in their desire to sec dirpct recommenda- 
tions. Those wishing the evaluatt>r to do theirjob for diem 
undoubtedly want reconunendations. However, this may be 
too strong a statement^ because many readen who want recom- 
mendations may deem that to be the responsibility of the evalu- 
ator and a completely proper and necessary function of evalu- 
ation. The counter argument, which has prevailed here, is that 
it is the responsibility of the program staff to know the options 
available and to recommend firom among them. Thus, we have 
almost always stopped short of level 5— recommending a 
specific action. 

The Executive Summary 

Executive summaries have been around for a long time. In 
fact, even in research, we have been vmting executive summa* 
ries for ages and calling them abstracts. However, we think 
that the big break for executive summaries came when evaluat- 
ors ^ . gan calling them executive summaries and conq^liment- 
ing the reader for being an executive. If you think this is too 
cynical, then the following reasons might be mem acceptable 
for why the executive summary is the key element of the 
effective evaluation report. 

1 . Deciiionmakers today eiuier do not have or are unwill- 
ing to invest the time to discover for themselves the 
key findings from an evaluation. 

2. Decisionmakers are more willing to delegate to the 
evaluator the identification of key fiindings than to take 
the time to do so themselves. 

3. The executive summary is an effective advance organ- 
izer for rtaders who need to be assisted 

4« The executive summary provides a well-thought-out 
set of summary statements that can be extracted for use 
in other reports or quoted. 



An effective executive summary 
needs to: 

• Have the ability to stand alone- 
lobe separated from the full report and be a 
useful, informative, independent document 

• Be only one page long. All right, 
maybe two pages are qjpropriate at times, 
but the shorter the better. 

• Include a program desc iption. 

• Have print big enough to read 
even though the evaluator held the length to 
one page. 

• Use buUets and highlights to lead 
the reader to key points. 
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The Evaluation Report 

So, we have moved^ over the past 15 years, away from the un- 
popdar and unread research-style evaluation report What 
were the features of those reports that provec so devastating? 

1. Length of the description of the 
study 

2. Lcngthofihe description of the 
analyses 

3. Length of the description of thv^ 
results 

4. Jargon and technical terminology 

5. Extensive tables 

6. Complex graphics 

7. Uninspiring layouts 

8. Uninspiring covers 

9. Uninspiring titles 

10* Convoluted sentence structure < 

11, ThIM-person construction 

Though the reader of this paper could translate these negative 
characteristics into the positive features of an effective evalu- 
ation report, good reporting calls for us to spell it out in a 
simple list 

1. Succinct description of the study 

2. Succinct description of the analy- 
ses—in an attachment rather than 
intermpting text 

3. Succinct description of the 
results 




4. Conversational language 



5. Simple presentation of numbers 
in tables 

6. Single idea tables that avoid nest- 
ing and interactions 

I. Moving complex tables to attach- 
ments at the end of the report 

8. Simple, single idea graphics 

9. "Popular" format like that in 
magazines and newspapers, with 
helpful headings 

10. Report covers that are interest- 
ing and/or related to the topic 

II. A descriptive or inspiring tide 
Utility of Familiar Featm^s 

When our audiences absolutely must deal with a stack of our 
evaluation reports, the !east we can do is try to facilitate their 
treks through them. Ihe best assistance we can offer is a con- 
sistent format that becomes familiar to them, allowing them to 
find their favorite sections or to skip over the sections they find 
less useful. 

In addition, helpful features such as boxed highlights, defini- 
tions/glossaries, and side bars all aid their interpretation and 
memory of key elements. 

Befores and Afters 

Attachments A and B contrast executive summaries and 
report covers respectively, before formats improved and after. 

Report Length 

The graphic at the right violates our premise that readers 
prefer single-idea, simple graphics. However, Ae audience for 
this paper is more sc^histocated and, if you have read this far 
already, certainly nxme motivated than most evaluation report 
readers. 



Report Titles 
Through the Years 

Final Evaluation Report (TITLE 
I ESEA) 1974-75 

1974-75 ESAA Pilot Project As- 
sist Teacher Attendance 
Formacive Report 

Analysis of the 1977-78 CAT 
Scores for SCE Identified 
Students 

Bilingual SyntaH Measure II - 
Comprehensiue English 
Language Test Equiuelency 
Study 1978-79 

1980-81 'Evacuation !fmdht£s 

Cable Chaiuiel 8: Is Austin Watching? 

When People Talk, AISD Listens! 
ORE Districtwide Surveys, 1983 

A Iflatter of Time: Retention and 
Promotion 

Promotion or Retention: 

Have Policies Passed or 
Failed? 

Haw Much Paperwork? A Survey 
ofPrtncipals' InBaskets, 1987 

'Mother Got Tired of Taking Care of 
My Baby^ A Study of Dropouts 
from AISD 
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Pages 

From 1974-75 through 1976-77, the length of our final K^ports 
grew dramatically. Then W.E. Jones wrote his letter. Since 
1977-78 our summary reports have been considerably shorter. 
From 1977-78 through 1981-82, all of our final reports for a 
year were completed July 1 and bound together. They were 
introduced by a 6 - 12 page summary. Beginning in 1982-83, 
we published separate finil reports with a separate sununary 
report for the year. The first three of these summary reports 
contained only l:ey findings; the next two years* publications 
included the executive summary from each evaluation. In 
1987-88, the need to publish findings before the adoption of the 
District budget inspired Preview, a limited-circulation look 
ahead at findings to be reported. 



Types of Final Reports 
and Findinj):^ Summaries 
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1974 No lummary document 

1975 No sununary document 

1976 No summary document 

1977 No summary document 
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Q 1978AtaGl&nce 

^ 1 0 pages, Key Fudingi 

I 1979 At a Glance 

1 2 pages, Key Findings 

I 1980 At a Glance 

§ 9 pages, Key Findings 

I 1981 At a Glance 

5 6 pages, Key Fmdings 
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1982 At a Glance 
6 pages, Key Findings 

ORE Evaluation Fmdings, IC'SS 
4 pages, Key Fmdings 

1984 At a Glance 

8 pages. Key Fmdings 

1985 At a Glance 

8 pages. Key Fmdings 

1986 At a Glance 

23 pages. Executive Summaries 

1987 At a Glance 
25 pigei, Executive 

1988 IVcvlcw 
4 pages, Key Findings 
19880vcfview 
40 pages. Exec Summ. 
ft OpcaLeitfln 



The New Open Letter 

A feature we tried this year with some success is the open 
letter* Evaluators had felt too restricted by our tight insistence 
that any conclusion or summary statement be substantially sup- 
ported by data. The open letter provided a forum for the 
evaluator to draw upon personal impressions and to make 
judgements about trends across time. As can be seen, the open 
letter format also allows freer discussion of options for pro- 
gram improvement. The sidebar on this page shows a sample 
of an open letter. 

Desktop Publishing 

Among other innovations, our office moved to word process- 
ing and is now using desktop publishing. In 1976-77 tiiere 
were final evaluation reports that not only were hundreds of 
pages long, but also originals that weighed double the weight 
of the same quantity of plain paper because of the taped and 
pasted pages. 

Word processing gave us the ability to correct, change, <tnd 
rearrange sections of reports, with only the graphs needing to 
be pasted in at the end. We could highlight, use italics, and 
underline at the touch of a keyboard. We could also catch 
typos - leaving us only "wordos," correctiy spelled incorrect 
words. There were fewer excuses for errors, and sometimes 
even fewer errors. 

Desktop publishing technology now allows us to gather word- 
processed text, computer designed graphics, data files, and 
practically anything else into a single document, use creative 
layouts, and print on a laser printer to produce documents that 
are near typeset quality. There is no longer any reason for an 
evaluation report to look like an evaluation report! 

So What? 

We would like to be able to report that all District evaluations 
are now read and valued by the staff who need the information. 
We would be pleased to report that there are no dull, homely 
looking, or repetitive reports. More realistically, we can report 
Uiat our publications are improved, that our readers are better 
informed, and that we continue to search for better ways to 
communicate technical information. 
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Open Letter to AISD 

TiUe Vn Evaluatton, 1987-88 

In oombinadon with other AISD prognms, TiQt VII 
appears to be working, especially based on long-term 
results. Of course, as Cummiit; (1985) points out, 
English-speaking classmates are not "standing still 
waiting for them to catch up." Especially in AISD, 
where average performance tends to be above the 
national average, Title VU musi enable students to 
**run harder and faster** to catch up and succeed. 
While Title VII does seem to be moving in this 
directioa the evaluation process did suggest some 
areas for possil^e improvement Readen are invited 
to draw their own impressions based on the dau in diis 
report and their own knowledge of the program. 

• Tutoring. National research h^ found that well- 
designed and implemented tutoring progr am s 
can be successful However, across the three 
years of Tide VII, positive eifects of the 
University of Texas tutors hav^ not been found. 
Students not tutored have shown patt^vns of 
growth similar to or greater than those of tutored 
students. The tutoring program appears to need 
revision. ... 

• ESLIoiDlDgt A total of 33 teachers in Tide 
vn schools, plus IS others, now have had ESL 
endorsement courses. Increased efforu to 
disseminate their names to appropriate school 
personnel could increase the number of LEP 
students scheduled into these classes. ... 

• Pflrgnt/Famtlv T: volvemgnt. Parent and family 
support groups provided through TiUe VII have 
begun to build a connection between the parents 
and the school....Child care, as provided at some 
meetings thia year, is a positive step. However, 
home visits, perhaps by ESL teachers, could 
reach parents who would not ordinarily attend 
workshops.... 

Thus, overall. Tide VII and AISD qipear to be making 
positive strides with these students. Continued 
refinements could result in an even more successful 
program. 

V J 
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AUnOM: HmmoT toraM. ratrlala topkl** 

About lisif mt mil slMMtarr r«taln««« m tlrm* tz^m •tudttnU. 
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lBpUMato4 prograM to oom oo oltorMtlvoo to rotootlon, 
Inoludiog two oir»Iuoto4 tklo f^r: 
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thoa to ooroil im olnth gro4o oourooo whllo ropootlng follo4 
olShth ero4o oowrooworh. 
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roUUooo o^ potoA- }«X rotaUooo ot Uo olMU Mhool/Juolor high 
loirol. lo oXoo doo«rlho4 hrlofir oM uood to oooporo «iu TAP. 
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9. Orada palat avaragaa. yoreantagaa iK^aolng oil oouroomk. 
dropout rataa. aad tha auahor ef olaaaaa failod for TAP 
otudonto oil ooaparod uaftvorahlr with full-raar r;italaooo. 
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for otudonto in ATT tha . for TAP otudonto. 

Whllo aoorlr oil otudonto 4npoor oueoooofullr to oooploto oil 
altamatlvaa atudlod. long-ttk o roaulta aro not ao poaltiva. 
Adjuataonti to thooo aodolo oovld laod to groatar aueooaa. teng- 
tara aupport aar ho noodod for thooo high-riok otudonta. 



'i4 



17 Jl 



No Pass— No Play: Impact on Fallurct, Dropouts, and Course EoroHments 
EXECUTIVE SUMMARY 

AUTHORt GlyoaUsoo 

In January, I9t5. At futei chanted. Slnoell»tUniB,8eoondaryftud«ii wbowUhtopaitidpaie 
in extTKurriculv actWiUti nusr pasi ALL coum each f U-wed^ 

iiy for die foUi»win|iU-waik period Iteooohoveny fumwndiost^ new rale focutad upon 
several key bniM. ThifnpoitfliMUegdiu ^vieiaaaiBy Impact the high echooUludettti In Ihe 
Aurthi I n d ep endent School DirtricL 



MAJOR nND04GS 



1 tftmrn ftf TM *^ p^'^ r''r 

bpeciaUj duriof lha fUl euiaeiv whM auny axtracunkulv acUvlttei ooour, die per- 
ontaia of UihiJKWl bUhig gnidei hie decthMd hon 15.5% In 19t4-tS 
1917-11 



XBUUdfl 



>Had h> a«l»acurH«»W m>— -7 Y.a. Hm 



decltaiB In fi^lfav gredei hat been ipaalv for fludentt who «« enrolled 
alad wkh axmcmkular actMttai. 

^^^ttidiiBMpifSdp^^ Forodier 
, tha dropflvt rato may be dadtatag. 

No. 

OvendU lha penMnUga of florolbaeaal diet are in honon counei hia nmain^ 
13%. gpowtag ftOB 13.6% to 13.9%. 



5. nu «huu>if »|foo Hit tti# ~i p«MMft pi 4v nile ancour oood them to F»»««r* fr^A^n 
Ym. to die flnt Ihne hi 1987-8S. a majority of AISO high tchool ttudenu (52%) agreed 
ihet die ao paic/ho play rule eoooungad them to BMke better gradei. 

On balance, the no pamlwo play rule eppeara to have been a poaitive change Clearly, 
becaim BO paMMo pUy began during a thna when many other changes wm being hnpleuMnted, 
we cannot ooochida wiihamurance that thiiooe rale change is re^KNuible for dwee poaitive 
outcomeaaMOciMwhhfoilhiggradeB.Aopouii.aodhoooriooumeorolhneata. However, 
a negative fanpactof no paasAio play hat failed to be evidenced in counecaroUmeoU oi overall 
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